Methodology
Dataset
A dataset of the number of daily reported total, confirmed COVID 19 cases were downloaded from The Humanitarian Data Exchange, a service provided by the United Nations Office for the Coordination of Humanitarian Affairs. Data on USA caseload was filtered and stored.
People mobility data was downloaded from Facebook Movement Range Maps, which contains measurement of movements compared to a baseline before social distancing and lock downs. Movement data for USA was filtered and was combined with daily reported number of cases. The combined dataset was split to obtain a 75-day assessment window and a training dataset for modeling.
Connection successful!
R is connected to the H2O cluster:
H2O cluster uptime: 14 hours 12 minutes
H2O cluster timezone: Asia/Colombo
H2O data parsing timezone: UTC
H2O cluster version: 3.32.1.5
H2O cluster version age: 11 days
H2O cluster name: H2O_started_from_R_Supun_iqy810
H2O cluster total nodes: 1
H2O cluster total memory: 0.68 GB
H2O cluster total cores: 4
H2O cluster allowed cores: 4
H2O cluster healthy: TRUE
H2O Connection ip: localhost
H2O Connection port: 54321
H2O Connection proxy: NA
H2O Internal Security: FALSE
H2O API Extensions: Amazon S3, Algos, AutoML, Core V3, TargetEncoder, Core V4
R Version: R version 3.6.3 (2020-02-29)
Modeling approach
The modeling pipeline was carried out in two pathways; incorporating mobility data, and without mobility data, in R language using open-source H20 machine learning libraries, creating and evaluating 6 time-series models and ensembles at each stage.
Stage 1 – Without mobility data
Original training set consists only the date as the predictor. The date has been subjected to various transformations to generate features like time lags, days, weeks, and holidays. Summary statistics of the 6 generated models are tabulated below.
Converting to H2OFrame...
Training H2O AutoML...
Leaderboard:
[7 rows x 6 columns]
Using top model: GBM_3_AutoML_20210816_094202
The best performing model is a Gradient Boosting Machine with 61 trees with depth of 8 levels and number of leaves ranging from 11 to 33 (mean of 23.22 leaves).
parsnip model object
Fit time: 36.8s
H2O AutoML - Gbm
--------
Model: Model Details:
==============
H2ORegressionModel: gbm
Model ID: GBM_3_AutoML_20210816_094202
Model Summary:
H2ORegressionMetrics: gbm
** Reported on training data. **
MSE: 56930353
RMSE: 7545.221
MAE: 3388.13
RMSLE: NaN
Mean Residual Deviance : 56930353
H2ORegressionMetrics: gbm
** Reported on cross-validation data. **
** 5-fold cross-validation on training data (Metrics computed for combined holdout predictions) **
MSE: 166250547
RMSE: 12893.82
MAE: 6204.571
RMSLE: NaN
Mean Residual Deviance : 166250547
Cross-Validation Metrics Summary:
The final model forecast for the 45-day assessment window and the next 6 months are as follow.
No Mobility data
[7 rows x 6 columns]
Stage II – With mobility data
In addition to transformed data regarding date and time, this model has two additional predictors; Change in Movement and Stay Put. Change in Movement looks at how much people are moving around and compares it with a baseline period that predates most social distancing measures, while Stay Put looks at the fraction of the population that appear to stay within a small area during an entire day.
The following graph shows how the Change in movement fluctuated over the time.
Variation of Stay Put is shown in the below graph
The effect of mobility on the epidemic curve can be considered to be ‘lagged’. In other words, the effect of change of people’s mobility pattern today will be reflected in the epidemic curve in the future.
Therefore we engineered lagged features from the original two mobility parameters to utilize as predictors for the time-series model. The maximum lag period was taken as one month; assuming if there is any effect of change in mobility over the caseload, it would be seen by one month from the day of mobility change.
In stage II, 6 models and ensembles were tuned and evaluated. Model metrics of them are summarized in the below table.
Converting to H2OFrame...
Training H2O AutoML...
Leaderboard:
[7 rows x 6 columns]
Using top model: StackedEnsemble_AllModels_AutoML_20210816_094319
The best performing model is a stacked ensemble of 1 Distributed Random Forest (DRF), 3 Gradient Boosting Machines (GBM) and 1 meta-learning Generalized Linear Model (GLM)
parsnip model object
Fit time: 30.4s
H2O AutoML - Stackedensemble
--------
Model: Model Details:
==============
H2ORegressionModel: stackedensemble
Model ID: StackedEnsemble_AllModels_AutoML_20210816_094319
Number of Base Models: 5
Base Models (count by algorithm type):
drf gbm glm
1 3 1
Metalearner:
Metalearner algorithm: glm
Metalearner cross-validation fold assignment:
Fold assignment scheme: AUTO
Number of folds: 5
Fold column: NULL
Metalearner hyperparameters:
H2ORegressionMetrics: stackedensemble
** Reported on training data. **
MSE: 8882627
RMSE: 2980.374
MAE: 1765.243
RMSLE: 0.0649412
Mean Residual Deviance : 8882627
H2ORegressionMetrics: stackedensemble
** Reported on cross-validation data. **
** 5-fold cross-validation on training data (Metrics computed for combined holdout predictions) **
MSE: 125620074
RMSE: 11208.04
MAE: 6046.679
RMSLE: NaN
Mean Residual Deviance : 125620074
Final model predictions for the assessment window are shown in the following graph.
With Mobility data
[7 rows x 6 columns]
The amount of mobility can be controlled for the future predictions by altering the mobility parameters in the feeding data frame. We simulated a low-mobility, stay-put situation for the next 6 months and the model’s predictions are as follows.
With min mobility
In a high mobility situation, the predictions are like this.
LS0tDQp0aXRsZTogIkNPVklEIDE5IFRpbWUgU2VyaWVzIEFuYWx5c2lzIC0gVVNBLiBXaXRoIGRpZmZlcmVudCBtb2JpbGl0aWVzIg0Kb3V0cHV0Og0KICBodG1sX25vdGVib29rOiBkZWZhdWx0DQogIGh0bWxfZG9jdW1lbnQ6DQogICAgZGZfcHJpbnQ6IHBhZ2VkDQogIHdvcmRfZG9jdW1lbnQ6IGRlZmF1bHQNCi0tLQ0KIyMjIE1ldGhvZG9sb2d5DQoNCiMjIyMgRGF0YXNldA0KDQpBIGRhdGFzZXQgb2YgdGhlIG51bWJlciBvZiBkYWlseSByZXBvcnRlZCB0b3RhbCwgY29uZmlybWVkIENPVklEIDE5IGNhc2VzIHdlcmUgZG93bmxvYWRlZCBmcm9tIFRoZSBIdW1hbml0YXJpYW4gRGF0YSBFeGNoYW5nZSwgYSBzZXJ2aWNlIHByb3ZpZGVkIGJ5IHRoZSBVbml0ZWQgTmF0aW9ucyBPZmZpY2UgZm9yIHRoZSBDb29yZGluYXRpb24gb2YgSHVtYW5pdGFyaWFuIEFmZmFpcnMuIERhdGEgb24gVVNBIGNhc2Vsb2FkIHdhcyBmaWx0ZXJlZCBhbmQgc3RvcmVkLg0KDQoNCg0KUGVvcGxlIG1vYmlsaXR5IGRhdGEgd2FzIGRvd25sb2FkZWQgZnJvbSBGYWNlYm9vayBNb3ZlbWVudCBSYW5nZSBNYXBzLCB3aGljaCBjb250YWlucyBtZWFzdXJlbWVudCBvZiBtb3ZlbWVudHMgY29tcGFyZWQgdG8gYSBiYXNlbGluZSBiZWZvcmUgc29jaWFsIGRpc3RhbmNpbmcgYW5kIGxvY2sgZG93bnMuIE1vdmVtZW50IGRhdGEgZm9yIFVTQSB3YXMgZmlsdGVyZWQgYW5kIHdhcyBjb21iaW5lZCB3aXRoIGRhaWx5IHJlcG9ydGVkIG51bWJlciBvZiBjYXNlcy4NClRoZSBjb21iaW5lZCBkYXRhc2V0IHdhcyBzcGxpdCB0byBvYnRhaW4gYSA3NS1kYXkgYXNzZXNzbWVudCB3aW5kb3cgYW5kIGEgdHJhaW5pbmcgZGF0YXNldCBmb3IgbW9kZWxpbmcuIA0KDQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGRwaSA9IDMwMCwgZmlnLnBhdGggPSAnZmlnLycsIGZpZy53aWR0aCA9IDgsIGZpZy5oZWlnaHQgPSA2LCBlY2hvID0gRkFMU0UsIGluY2x1ZGUgPSBUUlVFKQ0KDQpsaWJyYXJ5KHRpZHltb2RlbHMpDQpsaWJyYXJ5KG1vZGVsdGltZS5oMm8pDQpsaWJyYXJ5KHRpZHl2ZXJzZSkNCmxpYnJhcnkodGltZXRrKQ0KbGlicmFyeShyZWFkeGwpDQpsaWJyYXJ5KHJlYWRyKQ0KDQphc3Nfd2luZG93ID0gJzc1IGRheScNCmxhZ3NfZHVyYXJpb24gPSAxNQ0KbGFnc19zdHJldGNoID0gNw0KcHJlZF9ob3Jpem9uID0gJzYgbW9udGgnDQptaW5fbW9iX2FkYiA9IC0xDQptaW5fbW9iX2FyID0gMQ0KbWF4X21vYl9hZGIgPSAxDQptYXhfbW9iX2FyID0gMA0KDQpkMSA8LSByZWFkX2RlbGltKCJkYXRhL2ZiMSIsICJcdCIsIGVzY2FwZV9kb3VibGUgPSBGQUxTRSwgdHJpbV93cyA9IFRSVUUpDQpkMiA8LSByZWFkX2RlbGltKCJkYXRhL2ZiMiIsICJcdCIsIGVzY2FwZV9kb3VibGUgPSBGQUxTRSwgdHJpbV93cyA9IFRSVUUpDQoNCg0KdXMgPC0gZDEgJT4lIGZpbHRlcihjb3VudHJ5ID09ICdVU0EnKSAlPiUgcmJpbmQoZDIgJT4lIGZpbHRlcihjb3VudHJ5ID09ICdVU0EnKSkgJT4lIGdyb3VwX2J5KGRzKSAlPiUgc3VtbWFyaXNlKGFkYiA9IG1lYW4oYWxsX2RheV9iaW5nX3RpbGVzX3Zpc2l0ZWRfcmVsYXRpdmVfY2hhbmdlKSwgYXIgPSBtZWFuKGFsbF9kYXlfcmF0aW9fc2luZ2xlX3RpbGVfdXNlcnMpKSAlPiUgcmVuYW1lKERhdGUgPSBkcykNCg0KZ2IgPC0gcmVhZF9jc3YoImRhdGEvdGltZV9zZXJpZXNfY292aWQxOV9jb25maXJtZWRfZ2xvYmFsX25hcnJvdzIuY3N2IiwgIGNvbF90eXBlcyA9IGNvbHMoRGF0ZSA9IGNvbF9kYXRlKGZvcm1hdCA9ICIlbS8lZC8lWSIpKSkNCg0KDQojY3YgPC0gY3YgJT4lIGxlZnRfam9pbihsaykNCg0KIyNVUw0KY3YgPC0gZ2IgJT4lIGZpbHRlcihDb3VudHJ5ID09ICdVUycpICU+JSBzZWxlY3QoRGF0ZSwgbiA9IFZhbHVlKSAlPiUgbGVmdF9qb2luKHVzKSAlPiUgYXJyYW5nZShuKSAlPiUgbXV0YXRlKGxhZyA9IGxhZyhuLDEpKSAlPiUgbXV0YXRlKGN1bSA9IG4pICU+JSBtdXRhdGUobiA9IGN1bSAtIGxhZykgJT4lIHNlbGVjdChEYXRlLG4sYWRiLGFyKQ0KDQpjdjIgPC0gZ2IgJT4lIGZpbHRlcihDb3VudHJ5ID09ICdVUycpICU+JSBzZWxlY3QoRGF0ZSwgbiA9IFZhbHVlKSU+JSBhcnJhbmdlKG4pICU+JSBtdXRhdGUobGFnID0gbGFnKG4sMSkpICU+JSBtdXRhdGUoY3VtID0gbikgJT4lIG11dGF0ZShuID0gY3VtIC0gbGFnKSAlPiUgc2VsZWN0KERhdGUsIG4pDQoNCiMjIExhZ2dpbmcgZmVhdHVyZXMNCmN2IDwtIGN2ICU+JSB0a19hdWdtZW50X2xhZ3MoLiwudmFsdWUgPSBjKGFkYiwgYXIpLCAubGFncyA9IHNlcSgxLGxhZ3NfZHVyYXJpb24sIGJ5ID0gbGFnc19zdHJldGNoKSkNCmBgYA0KDQoNCmBgYHtyIGZpZy5oZWlnaHQ9NiwgZmlnLndpZHRoPTEwLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFLCBlY2hvPUZBTFNFfQ0KY3YgJT4lIHBsb3RfdGltZV9zZXJpZXMoLmRhdGVfdmFyID0gRGF0ZSwgbikNCiNjdiAlPiUgcGxvdF90aW1lX3NlcmllcyguZGF0ZV92YXIgPSBEYXRlLCBhZGIpDQojY3YgJT4lIHBsb3RfdGltZV9zZXJpZXMoLmRhdGVfdmFyID0gRGF0ZSwgYXIpDQoNCnNwbGl0cyA8LSB0aW1lX3Nlcmllc19zcGxpdChjdiwgYXNzZXNzID0gYXNzX3dpbmRvdywgY3VtdWxhdGl2ZSA9IFRSVUUpDQpzcGxpdHMyIDwtIHRpbWVfc2VyaWVzX3NwbGl0KGN2MiwgYXNzZXNzID0gYXNzX3dpbmRvdywgY3VtdWxhdGl2ZSA9IFRSVUUpDQoNCnJlY2lwZV9zcGVjIDwtIHJlY2lwZShuIH4gLiwgZGF0YSA9IHRyYWluaW5nKHNwbGl0cykpICU+JQ0KICAgIHN0ZXBfdGltZXNlcmllc19zaWduYXR1cmUoRGF0ZSkNCnJlY2lwZV9zcGVjMiA8LSByZWNpcGUobiB+IC4sIGRhdGEgPSB0cmFpbmluZyhzcGxpdHMyKSkgJT4lDQogICAgc3RlcF90aW1lc2VyaWVzX3NpZ25hdHVyZShEYXRlKQ0KDQp0cmFpbl90YmwgPC0gdHJhaW5pbmcoc3BsaXRzKSAlPiUgYmFrZShwcmVwKHJlY2lwZV9zcGVjKSwgLikNCnRlc3RfdGJsICA8LSB0ZXN0aW5nKHNwbGl0cykgJT4lIGJha2UocHJlcChyZWNpcGVfc3BlYyksIC4pDQoNCnRyYWluX3RibDIgPC0gdHJhaW5pbmcoc3BsaXRzMikgJT4lIGJha2UocHJlcChyZWNpcGVfc3BlYyksIC4pDQp0ZXN0X3RibDIgIDwtIHRlc3Rpbmcoc3BsaXRzMikgJT4lIGJha2UocHJlcChyZWNpcGVfc3BlYyksIC4pDQoNCg0KaDJvLmluaXQoDQogICAgbnRocmVhZHMgPSAtMSwNCiAgICBpcCAgICAgICA9ICdsb2NhbGhvc3QnLA0KICAgIHBvcnQgICAgID0gNTQzMjENCikNCmgyby5ub19wcm9ncmVzcygpDQptb2RlbF9zcGVjIDwtIGF1dG9tbF9yZWcobW9kZSA9ICdyZWdyZXNzaW9uJykgJT4lDQogICAgc2V0X2VuZ2luZSgNCiAgICAgICAgIGVuZ2luZSAgICAgICAgICAgICAgICAgICAgID0gJ2gybycsDQogICAgICAgICBtYXhfcnVudGltZV9zZWNzICAgICAgICAgICA9IDEwMCwgDQogICAgICAgICBtYXhfcnVudGltZV9zZWNzX3Blcl9tb2RlbCA9IDIwLA0KICAgICAgICAgbWF4X21vZGVscyAgICAgICAgICAgICAgICAgPSA1LA0KICAgICAgICAgbmZvbGRzICAgICAgICAgICAgICAgICAgICAgPSA1LA0KICAgICAgICAgZXhjbHVkZV9hbGdvcyAgICAgICAgICAgICAgPSBjKCksDQogICAgICAgICB2ZXJib3NpdHkgICAgICAgICAgICAgICAgICA9IDUsDQogICAgICAgICBzZWVkICAgICAgICAgICAgICAgICAgICAgICA9IDc4Ng0KICAgICkgDQpgYGANCg0KIyMjIyBNb2RlbGluZyBhcHByb2FjaA0KDQpUaGUgbW9kZWxpbmcgcGlwZWxpbmUgd2FzIGNhcnJpZWQgb3V0IGluIHR3byBwYXRod2F5czsgaW5jb3Jwb3JhdGluZyBtb2JpbGl0eSBkYXRhLCBhbmQgd2l0aG91dCBtb2JpbGl0eSBkYXRhLCBpbiBSIGxhbmd1YWdlIHVzaW5nIG9wZW4tc291cmNlIEgyMCBtYWNoaW5lIGxlYXJuaW5nIGxpYnJhcmllcywgY3JlYXRpbmcgYW5kIGV2YWx1YXRpbmcgNiB0aW1lLXNlcmllcyBtb2RlbHMgYW5kIGVuc2VtYmxlcyBhdCBlYWNoIHN0YWdlLg0KDQoNClN0YWdlIDEg4oCTIFdpdGhvdXQgbW9iaWxpdHkgZGF0YQ0KDQpPcmlnaW5hbCB0cmFpbmluZyBzZXQgY29uc2lzdHMgb25seSB0aGUgZGF0ZSBhcyB0aGUgcHJlZGljdG9yLiBUaGUgZGF0ZSBoYXMgYmVlbiBzdWJqZWN0ZWQgdG8gdmFyaW91cyB0cmFuc2Zvcm1hdGlvbnMgdG8gZ2VuZXJhdGUgZmVhdHVyZXMgbGlrZSB0aW1lIGxhZ3MsIGRheXMsIHdlZWtzLCBhbmQgaG9saWRheXMuIFN1bW1hcnkgc3RhdGlzdGljcyBvZiB0aGUgNiBnZW5lcmF0ZWQgbW9kZWxzIGFyZSB0YWJ1bGF0ZWQgYmVsb3cuIA0KDQoNCmBgYHtyfQ0KbW9kZWxfZml0dGVkMiA8LSBtb2RlbF9zcGVjICU+JQ0KICAgIGZpdChuIH4gLiwgZGF0YSA9IHRyYWluX3RibDIpDQpgYGANCg0KVGhlIGJlc3QgcGVyZm9ybWluZyBtb2RlbCBpcyBhIEdyYWRpZW50IEJvb3N0aW5nIE1hY2hpbmUgd2l0aCA2MSB0cmVlcyB3aXRoIGRlcHRoIG9mIDggbGV2ZWxzIGFuZCBudW1iZXIgb2YgbGVhdmVzIHJhbmdpbmcgZnJvbSAxMSB0byAzMyAobWVhbiBvZiAyMy4yMiBsZWF2ZXMpLg0KDQpgYGB7cn0NCm1vZGVsX2ZpdHRlZDINCmBgYA0KDQoNClRoZSBmaW5hbCBtb2RlbCBmb3JlY2FzdCBmb3IgdGhlIDQ1LWRheSBhc3Nlc3NtZW50IHdpbmRvdyBhbmQgdGhlIG5leHQgNiBtb250aHMgYXJlIGFzIGZvbGxvdy4NCg0KDQoNCiMjIyMgTm8gTW9iaWxpdHkgZGF0YQ0KYGBge3IgZmlnLmhlaWdodD02LCBmaWcud2lkdGg9MTAsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9DQptb2RlbHRpbWVfdGJsIDwtIG1vZGVsdGltZV90YWJsZSgNCiAgICBtb2RlbF9maXR0ZWQyDQopIA0KDQptb2RlbHRpbWVfdGJsICU+JQ0KICBtb2RlbHRpbWVfY2FsaWJyYXRlKHRlc3RfdGJsMikgJT4lDQogICAgbW9kZWx0aW1lX2ZvcmVjYXN0KA0KICAgICAgICBuZXdfZGF0YSAgICA9IHRlc3RfdGJsMiwNCiAgICAgICAgYWN0dWFsX2RhdGEgPSBjdjIsDQogICAgICAgIGtlZXBfZGF0YSAgID0gVFJVRQ0KICAgICkgJT4lDQogICAgcGxvdF9tb2RlbHRpbWVfZm9yZWNhc3QoDQogICAgICAgIC5pbnRlcmFjdGl2ZSA9IFRSVUUNCiAgICApDQoNCmRhdGFfcHJlcGFyZWRfdGJsIDwtIGJpbmRfcm93cyh0cmFpbl90YmwyLCB0ZXN0X3RibDIpDQoNCmZ1dHVyZV90YmwgPC0gZGF0YV9wcmVwYXJlZF90YmwgJT4lDQogICAgZnV0dXJlX2ZyYW1lKC5sZW5ndGhfb3V0ID0gcHJlZF9ob3Jpem9uKQ0KDQpmdXR1cmVfcHJlcGFyZWRfdGJsIDwtIGJha2UocHJlcChyZWNpcGVfc3BlYyksIGZ1dHVyZV90YmwpDQoNCnJlZml0X3RibCA8LSBtb2RlbHRpbWVfdGJsICU+JQ0KICAgIG1vZGVsdGltZV9yZWZpdChkYXRhX3ByZXBhcmVkX3RibCkNCg0KcmVmaXRfdGJsICU+JQ0KICAgIG1vZGVsdGltZV9mb3JlY2FzdCgNCiAgICAgICAgbmV3X2RhdGEgICAgPSBmdXR1cmVfcHJlcGFyZWRfdGJsLA0KICAgICAgICBhY3R1YWxfZGF0YSA9IGRhdGFfcHJlcGFyZWRfdGJsLA0KICAgICAgICBrZWVwX2RhdGEgICA9IFRSVUUNCiAgICApICU+JQ0KICAgIHBsb3RfbW9kZWx0aW1lX2ZvcmVjYXN0KA0KICAgICAgICAuaW50ZXJhY3RpdmUgPSBUUlVFLC5jb25mX2ludGVydmFsX3Nob3cgPSBUUlVFDQogICAgKQ0KYGBgDQoNCg0KDQoNCg0KIyMjIyBTdGFnZSBJSSDigJMgV2l0aCBtb2JpbGl0eSBkYXRhDQoNCkluIGFkZGl0aW9uIHRvIHRyYW5zZm9ybWVkIGRhdGEgcmVnYXJkaW5nIGRhdGUgYW5kIHRpbWUsIHRoaXMgbW9kZWwgaGFzIHR3byBhZGRpdGlvbmFsIHByZWRpY3RvcnM7IENoYW5nZSBpbiBNb3ZlbWVudCBhbmQgU3RheSBQdXQuIENoYW5nZSBpbiBNb3ZlbWVudCBsb29rcyBhdCBob3cgbXVjaCBwZW9wbGUgYXJlIG1vdmluZyBhcm91bmQgYW5kIGNvbXBhcmVzIGl0IHdpdGggYSBiYXNlbGluZSBwZXJpb2QgdGhhdCBwcmVkYXRlcyBtb3N0IHNvY2lhbCBkaXN0YW5jaW5nIG1lYXN1cmVzLCB3aGlsZSBTdGF5IFB1dCBsb29rcyBhdCB0aGUgZnJhY3Rpb24gb2YgdGhlIHBvcHVsYXRpb24gdGhhdCBhcHBlYXIgdG8gc3RheSB3aXRoaW4gYSBzbWFsbCBhcmVhIGR1cmluZyBhbiBlbnRpcmUgZGF5Lg0KDQoNClRoZSBmb2xsb3dpbmcgZ3JhcGggc2hvd3MgaG93IHRoZSBDaGFuZ2UgaW4gbW92ZW1lbnQgZmx1Y3R1YXRlZCBvdmVyIHRoZSB0aW1lLiANCg0KYGBge3J9DQpjdiAlPiUgcGxvdF90aW1lX3NlcmllcyguZGF0ZV92YXIgPSBEYXRlLCBhZGIpDQpgYGANCg0KVmFyaWF0aW9uIG9mIFN0YXkgUHV0IGlzIHNob3duIGluIHRoZSBiZWxvdyBncmFwaA0KDQpgYGB7cn0NCmN2ICU+JSBwbG90X3RpbWVfc2VyaWVzKC5kYXRlX3ZhciA9IERhdGUsIGFyKQ0KYGBgDQoNClRoZSBlZmZlY3Qgb2YgbW9iaWxpdHkgb24gdGhlIGVwaWRlbWljIGN1cnZlIGNhbiBiZSBjb25zaWRlcmVkIHRvIGJlIOKAmGxhZ2dlZOKAmS4gSW4gb3RoZXIgd29yZHMsIHRoZSBlZmZlY3Qgb2YgY2hhbmdlIG9mIHBlb3BsZeKAmXMgbW9iaWxpdHkgcGF0dGVybiB0b2RheSB3aWxsIGJlIHJlZmxlY3RlZCBpbiB0aGUgZXBpZGVtaWMgY3VydmUgaW4gdGhlIGZ1dHVyZS4NCg0KVGhlcmVmb3JlIHdlIGVuZ2luZWVyZWQgbGFnZ2VkIGZlYXR1cmVzIGZyb20gdGhlIG9yaWdpbmFsIHR3byBtb2JpbGl0eSBwYXJhbWV0ZXJzIHRvIHV0aWxpemUgYXMgcHJlZGljdG9ycyBmb3IgdGhlIHRpbWUtc2VyaWVzIG1vZGVsLiBUaGUgbWF4aW11bSBsYWcgcGVyaW9kIHdhcyB0YWtlbiBhcyBvbmUgbW9udGg7IGFzc3VtaW5nIGlmIHRoZXJlIGlzIGFueSBlZmZlY3Qgb2YgY2hhbmdlIGluIG1vYmlsaXR5IG92ZXIgdGhlIGNhc2Vsb2FkLCBpdCB3b3VsZCBiZSBzZWVuIGJ5IG9uZSBtb250aCBmcm9tIHRoZSBkYXkgb2YgbW9iaWxpdHkgY2hhbmdlLg0KDQpJbiBzdGFnZSBJSSwgNiBtb2RlbHMgYW5kIGVuc2VtYmxlcyB3ZXJlIHR1bmVkIGFuZCBldmFsdWF0ZWQuIE1vZGVsIG1ldHJpY3Mgb2YgdGhlbSBhcmUgc3VtbWFyaXplZCBpbiB0aGUgYmVsb3cgdGFibGUuDQoNCmBgYHtyfQ0KbW9kZWxfZml0dGVkIDwtIG1vZGVsX3NwZWMgJT4lDQogICAgZml0KG4gfiAuLCBkYXRhID0gdHJhaW5fdGJsKQ0KYGBgDQoNClRoZSBiZXN0IHBlcmZvcm1pbmcgbW9kZWwgaXMgYSBzdGFja2VkIGVuc2VtYmxlIG9mIDEgRGlzdHJpYnV0ZWQgUmFuZG9tIEZvcmVzdCAoRFJGKSwgMyBHcmFkaWVudCBCb29zdGluZyBNYWNoaW5lcyAoR0JNKSBhbmQgMSBtZXRhLWxlYXJuaW5nIEdlbmVyYWxpemVkIExpbmVhciBNb2RlbCAoR0xNKQ0KDQpgYGB7cn0NCm1vZGVsX2ZpdHRlZA0KYGBgDQoNCg0KRmluYWwgbW9kZWwgcHJlZGljdGlvbnMgZm9yIHRoZSBhc3Nlc3NtZW50IHdpbmRvdyBhcmUgc2hvd24gaW4gdGhlIGZvbGxvd2luZyBncmFwaC4NCg0KDQojIyMjIFdpdGggTW9iaWxpdHkgZGF0YQ0KDQpgYGB7ciBmaWcuaGVpZ2h0PTYsIGZpZy53aWR0aD0xMCwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0NCm1vZGVsdGltZV90YmwgPC0gbW9kZWx0aW1lX3RhYmxlKA0KICAgIG1vZGVsX2ZpdHRlZA0KKSANCg0KbW9kZWx0aW1lX3RibCAlPiUNCiAgbW9kZWx0aW1lX2NhbGlicmF0ZSh0ZXN0X3RibCkgJT4lDQogICAgbW9kZWx0aW1lX2ZvcmVjYXN0KA0KICAgICAgICBuZXdfZGF0YSAgICA9IHRlc3RfdGJsLA0KICAgICAgICBhY3R1YWxfZGF0YSA9IGN2LA0KICAgICAgICBrZWVwX2RhdGEgICA9IFRSVUUNCiAgICApICU+JQ0KICAgIHBsb3RfbW9kZWx0aW1lX2ZvcmVjYXN0KA0KICAgICAgICAuaW50ZXJhY3RpdmUgPSBUUlVFDQogICAgKQ0KDQpkYXRhX3ByZXBhcmVkX3RibCA8LSBiaW5kX3Jvd3ModHJhaW5fdGJsLCB0ZXN0X3RibCkNCg0KcmVmaXRfdGJsIDwtIG1vZGVsdGltZV90YmwgJT4lDQogICAgbW9kZWx0aW1lX3JlZml0KGRhdGFfcHJlcGFyZWRfdGJsKQ0KYGBgDQoNCg0KVGhlIGFtb3VudCBvZiBtb2JpbGl0eSBjYW4gYmUgY29udHJvbGxlZCBmb3IgdGhlIGZ1dHVyZSBwcmVkaWN0aW9ucyBieSBhbHRlcmluZyB0aGUgbW9iaWxpdHkgcGFyYW1ldGVycyBpbiB0aGUgZmVlZGluZyBkYXRhIGZyYW1lLiBXZSBzaW11bGF0ZWQgYSBsb3ctbW9iaWxpdHksIHN0YXktcHV0IHNpdHVhdGlvbiBmb3IgdGhlIG5leHQgNiBtb250aHMgYW5kIHRoZSBtb2RlbOKAmXMgcHJlZGljdGlvbnMgYXJlIGFzIGZvbGxvd3MuDQoNCg0KIyMjIyBXaXRoIG1pbiBtb2JpbGl0eQ0KDQpgYGB7ciBmaWcuaGVpZ2h0PTYsIGZpZy53aWR0aD0xMCwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0NCmZ1dHVyZV90YmwgPC0gZGF0YV9wcmVwYXJlZF90YmwgJT4lDQogICAgZnV0dXJlX2ZyYW1lKC5sZW5ndGhfb3V0ID0gcHJlZF9ob3Jpem9uKQ0KDQpmdXR1cmVfcHJlcGFyZWRfdGJsIDwtIGJha2UocHJlcChyZWNpcGVfc3BlYyksIGZ1dHVyZV90YmwpICU+JSBtdXRhdGUoYWRiID0gbWluX21vYl9hZGIsIGFyID0gbWluX21vYl9hcikgJT4lIHRrX2F1Z21lbnRfbGFncyguLC52YWx1ZSA9IGMoYWRiLCBhciksIC5sYWdzID0gc2VxKDEsbGFnc19kdXJhcmlvbiwgYnkgPSBsYWdzX3N0cmV0Y2gpKQ0KDQoNCnJlZml0X3RibCAlPiUNCiAgICBtb2RlbHRpbWVfZm9yZWNhc3QoDQogICAgICAgIG5ld19kYXRhICAgID0gZnV0dXJlX3ByZXBhcmVkX3RibCwNCiAgICAgICAgYWN0dWFsX2RhdGEgPSBkYXRhX3ByZXBhcmVkX3RibCwNCiAgICAgICAga2VlcF9kYXRhICAgPSBUUlVFDQogICAgKSAlPiUNCiAgICBwbG90X21vZGVsdGltZV9mb3JlY2FzdCgNCiAgICAgICAgLmludGVyYWN0aXZlID0gVFJVRSwuY29uZl9pbnRlcnZhbF9zaG93ID0gVFJVRQ0KICAgICkNCmBgYA0KDQpJbiBhIGhpZ2ggbW9iaWxpdHkgc2l0dWF0aW9uLCB0aGUgcHJlZGljdGlvbnMgYXJlIGxpa2UgdGhpcy4NCg0KIyMjIyBXaXRoIG1heCBtb2JpbGl0eQ0KDQpgYGB7ciBmaWcuaGVpZ2h0PTYsIGZpZy53aWR0aD0xMCwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0NCmZ1dHVyZV9wcmVwYXJlZF90YmwgPC0gYmFrZShwcmVwKHJlY2lwZV9zcGVjKSwgZnV0dXJlX3RibCkgJT4lIG11dGF0ZShhZGIgPSBtYXhfbW9iX2FkYiwgYXIgPSBtYXhfbW9iX2FyKSAlPiUgdGtfYXVnbWVudF9sYWdzKC4sLnZhbHVlID0gYyhhZGIsIGFyKSwgLmxhZ3MgPSBzZXEoMSxsYWdzX2R1cmFyaW9uLCBieSA9IGxhZ3Nfc3RyZXRjaCkpDQoNCnJlZml0X3RibCAlPiUNCiAgICBtb2RlbHRpbWVfZm9yZWNhc3QoDQogICAgICAgIG5ld19kYXRhICAgID0gZnV0dXJlX3ByZXBhcmVkX3RibCwNCiAgICAgICAgYWN0dWFsX2RhdGEgPSBkYXRhX3ByZXBhcmVkX3RibCwNCiAgICAgICAga2VlcF9kYXRhICAgPSBUUlVFDQogICAgKSAlPiUNCiAgICBwbG90X21vZGVsdGltZV9mb3JlY2FzdCgNCiAgICAgICAgLmludGVyYWN0aXZlID0gVFJVRSwuY29uZl9pbnRlcnZhbF9zaG93ID0gVFJVRQ0KICAgICkNCg0KYGBgDQoNCiMjIyBDb21wYXJpc29uIG9mIHRoZSB0d28gZmluYWwgbW9kZWxzOyBpcyBtb2JpbGl0eSBkYXRhIHRydWx5IGluZm9ybWF0aXZlPw0KDQpUaGUgZm9sbG93aW5nIGdyYXBoIGNvbXBhcmVzIHRoZSBwcmVkaWN0aW9uIGVycm9yIG1hZGUgYnkgdGhlIHR3byBtb2RlbHM7IHdpdGhvdXQgbW9iaWxpdHkgZGF0YSBhbmQgd2l0aCBtb2JpbGl0eSBkYXRhLg0KDQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0Kc3VtIDwtIHJlYWRfY3N2KCJkYXRhL3N1bS5jc3YiKQ0Kc3VtICU+JSBwaXZvdF9sb25nZXIoY29scyA9ICdSTVNFJzonTUFFX2N2JykgJT4lIGdncGxvdChhZXMoeCA9IG5hbWUsIHkgPSB2YWx1ZSwgY29sID0gRXhwZXJpbWVudCwgZmlsbCA9IEV4cGVyaW1lbnQpKSArIGdlb21fY29sKHBvc2l0aW9uID0gJ2RvZGdlJykgKyB0aGVtZV9idygpICsgeGxhYignTWV0cmljJykNCmBgYA0KDQpJdCBjYW4gYmUgY2xlYXJseSBzZWVuIHRoYXQgdGhlIGVycm9yIGZvciBib3RoIHRyYWluaW5nIGRhdGEgYW5kIGNvbWJpbmVkIGhvbGRvdXQgY3Jvc3MtdmFsaWRhdGlvbiBzYW1wbGVzIGlzIGxlc3MgaW4gdGhlIG1vZGVsIHdpdGggbW9iaWxpdHkgZGF0YS4gSW4gb3RoZXIgd29yZHMsIHRoZSBtb2JpbGl0eSBwYXJhbWV0ZXJzIGFjdHVhbGx5IHBsYXkgYSByb2xlIGluIHNoYXBpbmcgdGhlIGVwaWRlbWljIGN1cnZlLiBJdCBjYW4gYWxzbyBiZSBzZWVuIGJ5IG9ic2VydmluZyB0aGUgdHdvIHNpbXVsYXRlZCBmdXR1cmUgNi1tb250aCBwZXJpb2RzLiBUaGUgbG93LW1vYmlsaXR5IHN0YXktcHV0IHNpdHVhdGlvbiBpcyBmb3JlY2FzdGVkIHRvIGhhdmUgYSByZWxhdGl2ZWx5IGxvdyB0b3RhbCBudW1iZXIgb2YgaW5mZWN0ZWQgcGVvcGxlIHRoYW4gdGhlIGhpZ2ggbW9iaWxpdHkgcGVyaW9kLiANCg0KIyMjIyBXaGF0IGlzIG5ldz8NCg0KIyMjIyMgVGhpcyBleHBlcmltZW50IHdhcyBub3QgYmFzZWQgb24gZXBpZGVtaW9sb2dpY2FsIGRpc2Vhc2UgbW9kZWxzIHRvIGRlc2NyaWJlIGFuZCBmb3JlY2FzdCB0aGUgdHJhamVjdG9yeSBvZiB0aGUgY2FzZWxvYWQsIGJ1dCB3YXMgcHVyZWx5IGRhdGEtZHJpdmVuLiBXZSB0cmllZCB0byB1dGlsaXplIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtcyB0byBpZGVudGlmeSBwb3RlbnRpYWwgcHJlZGljdG9ycyBhbmQgdG8gbGVhcm4gdGhlIGJlc3QgbW9kZWwgcGFyYW1ldGVycyB3aGljaCBtaW5pbWl6ZWQgdGhlIGVycm9yLCBvciBpbiBvdGhlciB3b3JkcyB3aGljaCBmaXQgYmVzdCB0byB0aGUgb2JzZXJ2ZWQgZGF0YS4gDQoNCg0K